System and method for estimating/determining the date of a photo

ABSTRACT

Undated photos are organized by estimating the date of each photo. The date is estimated by building a model based on a set of reference photos having established dates, and comparing image characteristics of the undated photo to the image characteristics of the reference photos. The photo characteristics can include hues, saturation, intensity, contrast, sharpness and graininess as represented by image pixel data. Once the date of a photo is estimated, it can be tagged with identifying information, such as by using the estimated date to associate the photo with a node in a family tree.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.13/440,754, filed Apr. 5, 2012, which is herein incorporated byreference in its entirety for all purposes.

BACKGROUND OF THE INVENTION

With the widespread use of digital photography and digital storage,people are collecting large numbers of photos and copies of photos. Insome instances, the photos may be hardcopy or hard print photos (storedin shoe boxes, photo albums and so forth) that were taken many yearsbefore, or may be soft, digitally stored copies of hard print photosthat have been digitally scanned. While photos might be organized byeither placing them in electronic folders or by tagging the photos(i.e., digitally labeling or identifying the photos with pertinentinformation, such as who is in the photo, where the photo was taken, andwhen it was taken), this can be very difficult because of the largenumber of photos that most people collect, because the amount of timethat it takes to individually view and classify or identify each photo,and in some cases, because of the age of the photo or other factors thatmake it difficult to accurately identify the photo.

In some cases, when trying to identify old family photos, people mayreference genealogical information, such as a family tree, and guess asto which person in a family tree the photo may relate to. But withoutknowing the date that the photo was taken, using a family tree toidentify a photo can be difficult.

BRIEF SUMMARY OF THE INVENTION

There is provided, in accordance with embodiments of the presentinvention, a system and method for estimating the image capture date ofa photo. The estimated date is based on a comparison of imagecharacteristics of the photo to the image characteristics of referencephotos that each have an established or known date.

In one embodiment, the system and method identify image characteristicsof an undated photo, compare the image characteristics of the undatedphoto to image characteristics derived from a plurality of referencephotos, where each of the plurality of reference photos have anestablished date, and estimate the date of the undated photo based onthe similarity of the image characteristics of the undated photo to oneor more of the image characteristics of the reference photos. In adescribed embodiment, the image characteristics are obtained fromprocessed pixel data, and include hues, saturation, intensity, contrast,sharpness and graininess.

In another embodiment of the invention, a method and system are providedfor identifying a photo. The photo is displayed at a user interface. Theimage capture date of the displayed photo is estimated by identifyingimage characteristics of the displayed photo from pixel datarepresenting the image in the displayed photo, comparing the imagecharacteristics of the displayed photo to image characteristics from aplurality of reference photos, each of the plurality of reference photoshaving an established date, and estimating the date of the displayedphoto based on the similarity of the image characteristics of thedisplayed photo to one or more the image characteristics of thereference photos. A data structure (such as a family tree) having aplurality of data nodes, each data node representing one or more datarecords having a subject associated with one or more dates, is displayedat the user interface. The system and method determine whether there isa relationship between the displayed photo and a data record at one ormore of the data nodes, based on the estimated date of the displayedphoto, identify at the displayed data structure one or more data nodesthat have a relationship to the displayed photo based on the datesassociated with the subject of the one or more data nodes, and receivetagging information at a user interface concerning the displayed photobased on the identified one or more data nodes.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the present invention may be derived byreferring to the detailed description of the invention and to theclaims, when considered in connection with the Figures.

FIG. 1 illustrates identifying and organizing a set of photos inaccordance with an embodiment of the invention.

FIG. 2 is a simplified flow diagram illustrating a method for estimatingthe date of photographs in accordance with one embodiment of theinvention.

FIG. 3 illustrates a date estimation model or rules engine that could beused in the process of FIG. 2.

FIGS. 4A through 4D illustrate exemplary image characteristics ofphotographs that could be used in developing the date estimation modelseen in FIG. 3.

FIG. 5 is a detailed flow diagram illustrating a method for estimatingthe date of photographs and using the estimated date to tag photographs,in accordance with an embodiment of the invention.

FIG. 6 is a block diagram of a system for estimating the date ofphotographs and tagging those photographs with identifying information.

FIG. 7 illustrates a display on a user interface, used in implementingthe process of FIG. 5.

FIG. 8 is a flow diagram illustrating the use of pixel datacharacteristics in estimating the date of a photograph.

FIG. 9 is a block diagram illustrating an exemplary computer system uponwhich embodiments of the present invention may be implemented.

DETAILED DESCRIPTION OF THE INVENTION

There are various embodiments and configurations for implementing thepresent invention. Generally, embodiments provide systems and methodsfor estimating the date of a photograph (the date that the image in thephotograph was captured). By estimating the date of a photograph,identifying data for the photograph may be inferred, e.g., by a userviewing the photograph.

In one embodiment, a photo can have its date estimated by comparing itsimage characteristics to image characteristics of a collection ofreference photos. The reference photographs have established dates,permitting a model to be built that associates various imagecharacteristics to the established dates of the reference photos. In onecase, image characteristics include image characteristics taken fromprocessed pixel data, such as image data reflecting the hues,saturation, intensity, contrast, sharpness and graininess in theunderlying photo (and its image). Some characteristics (e.g., hues,saturation and intensity) are taken from color space or model data, suchas the HSI (hue, saturation and intensity) color model data. Somecharacteristics (sharpness and graininess) can be taken from frequencydomain data, e.g., pixel data that has been converted into a Fourierseries and using frequency components to establish the degree ofsharpness and graininess.

In some embodiments, the image characteristics of a large sampling orcollection of photographs with known dates can be determined in order tobuild a model. In a specific embodiment, the collection of photographsare from data records stored in connection with genealogical records. Inparticular, users of a genealogical data can set up family trees, witheach person or node in the trees having records, such as photographs,stored for that person. Such records may include tagging informationwhich has been entered by a person familiar with the family tree, withthe tagging information including the date that a reference photo wastaken. As a result of many people storing records (includingphotographs) for many family trees, the stored photographs are likely tobe numerous and also span across many years. By processing the imagecharacteristics for the stored photographs, a model can be built whichcan be used to estimate the date of a photograph based on the imagecharacteristics of the reference photographs and dates that are known orestablished for those reference photographs. In some cases, theestimated date may be a timeframe or a date range (e.g., “January,1920,” “between Jan. 1, 1970 and Jan. 1, 1975,” or “on or before Jan. 1,1990”).

Once the date of a photo has been estimated (using the model), a usercan then enter identifying (tagging) information for the photo, if suchdata can be inferred or determined based on the estimated date. Forexample, in one embodiment, a user can view a displayed photograph and adata structure, such as a family tree, on a user interface. Based on theestimated date of the displayed photo, the user (who would typically befamiliar with or may have created the family tree), may be able todetermine the identity of a person (or people) in the photograph basedon the estimated date and the people in the family tree that were aliveon the estimated date.

Thus, an estimated date of a photograph can be used either to generallyorganize photos by estimated dates (e.g., arrange the photos byestimated date or time frame), or to more specifically identify a photoby providing context to the photo and facilitate or aid a user in addinginformation on a person (name), place, and/or event to which the photopertains. With such information or tags, photos can later be retrievedbased on search terms relating to people, places or events.

In other embodiments, basic image information (rather than processedpixel data) can also be used to estimate or determine the date of aphotograph. For example, image data such as aspect ratio and type ofphoto border could be used. As another example, metadata stored with adigital photograph (date, type of camera, etc.) could be accessed andalso used to estimate the date of a photo.

In all these various embodiments, the estimation or determination of thedate of a photo can be used by a user to facilitate the entry ofidentifying information in order to organize photos and/or make themaccessible based on identifying information.

Referring now to FIG. 1, there is illustrated a set of untagged photosor images 100 that can be identified and organized according to featuresof the present invention. The photos 100 are not organized (oridentified), and can relate to numerous people, events, places or thingsfor which a user may not have information or other means to identify thephotos. In some instances the photos may be hardcopy prints that mayhave been stored in a box or album, and may have many different formats,styles and sizes (e.g., from having been taken over many years). As willbe more fully described below, if the photos are hardcopies, the photosare scanned and digitally stored. In some instances, the user may havephotos already in digital form, either because they have been previouslyscanned or they may have been originally taken with a digital camera anddownloaded to a storage device.

In accordance with some embodiments herein, information concerning thephotos is obtained so that the images can be identified and stored ascategorized or organized images 110. It should be appreciated that bybeing organized, the photos are not necessarily stored together, butrather are classified or “tagged” with identifiers. As illustrated inFIG. 1, tagged photos might be classified or organized into groups 120.In the illustrated embodiment, the photos are organized by dates ortimeframes, e.g., photos in twenty year time frames are categorizedtogether in groups identified by the years 1890, 1910, 1930, 1950, etc.However, as should be appreciated, these are only examples and,depending on the confidence in date estimation, photos could beorganized by individual years or perhaps even months. Further, whilephotos are shown in FIG. 1 as organized by dates, as will be discussedlater, in some embodiments, date estimation is used as a first step intagging photos with other identifying information (a person in thephoto, where it was taken, when it was taken, or an event that itcaptures), and in such a way the photos can be retrieved and viewed byusing those other identifiers. As an example, a user might want to viewall photographs concerning his or her grandfather and, using thegrandfather's name as an identifier, retrieve all digitally photos thathave been tagged with the grandfather's name. Thus, each of the groupsof photos 120 may have a common tag or identifier (such as a timeframe).However, other tags (people, place, event, etc.) could be used toorganize the photos. Further, in some instances, a photo may havemultiple tags (for example, multiple people, events, locations or acombination thereof) and one photo may be assigned to or classified inmore than one of categories in the groups 120.

While the described embodiments relate to tagging still photos (asillustrated in FIG. 1), it should be appreciated that the invention hasapplication to other forms of images, such as motion pictures andvideos.

Referring now to FIG. 2, there is seen a simplified flow diagramillustrating a process for estimating the date of a photograph inaccordance with one embodiment of the invention. At step 210, a model orrules engine is developed based on a collection of reference photographswith known or established dates.

As mentioned earlier, the reference photographs might be those stored inconnection with genealogical records, in order to provide a largesampling of photographs over many years. For example, photographs storedin database systems that permit users to develop and maintain familytrees (and to store photographs and other records), such as systemsoperated by Ancestry.com (Provo, Utah), could be used. Such databasesystems might store hundreds of thousands (if not millions) ofphotographs that have been provided in digital form by users of thosesystems, with the photographs stored at nodes representing people andhaving tagging information that would typically include a date (orapproximate date) that the photograph was taken (along with otherinformation, such as people in the photograph, location, event, and soforth). Photographs stored in connection with genealogical informationand family trees are particularly useful for purposes of developing amodel since the photos would typically extend across many years as partof family trees (from the time of the earliest photographs in themid-1800s up to the present time). However, other databases could beused as long as the sampling of photos is sufficiently large.

In connection with step 210, two general types of image characteristicsare identified from the reference photographs, namely (1) basiccharacteristics of the image and (2) characteristics from processingpixel data in the image. These characteristics are illustrated in FIG.3, which show various characteristics that are built into a dateestimation model or rules engine 300.

The first type of general image characteristics (basic imagecharacteristics) include image border characteristics that areillustrated by the rules category 310, and aspect ratio characteristicsthat are illustrated by the rules category 312. As to image bordercharacteristics, borders and border styles have evolved over timedepending on the type of camera and the photo finishing process used.White borders were popular in the past, with varying border sizes andshapes that often depended on the type of film or photofinishing processused during the timeframe that the photograph was taken. In addition,and during certain periods of time, scalloped borders and other bordereffects were popular on print photos. White borders began to disappearaltogether as digital photography became popular.

Aspect ratios (relationship of width to height of a camera image) havealso evolved over time, and often depend on the type of film used (priorto digital photography). For example, INSTAMATIC cameras used in the1960s produced 3.5 inch×3.5 prints. Cameras popular in the 1970s usedfilm cartridges that produced 3.5″×5″ prints.

The second type of general image characteristics relates to the imagepixels that result from digitally processing an image. As illustrated inFIG. 3, rules can be developed for hues or colors (rules category 320),saturation (rules category 322), intensity (rules category 324),contrast (rules category 332), sharpness or resolution (rules category334) and graininess (rules category 336). Further details on processingand identifying these characteristics will be provided later inconnection with FIG. 8.

The various characteristics are used to develop the model 300 (FIG. 3)which can then be later used to estimate the date of a given photo basedon that photo's characteristics in relation to characteristics (builtinto model 300) of references photos and their known dates.

FIGS. 4A through 4D show exemplary photographic characteristics overtime in order to illustrate how a model, such as the date estimationmodel 300 in FIG. 3, could be built using the various image pixelcharacteristics 320-324 and 332-336 illustrated in FIG. 3.

In FIG. 4A, there is a graph illustrating the presence of colors (beyondblack and white) in photos from 1900 to 2010. As illustrated, some verysmall amounts of certain colors (yellow and sepia tones) appear inphotos taken prior to the 1960s, as a result of aging and exposure ofblack and white print photos to the environment (such colors aregreatest in the oldest photos). Color photographs taken by consumers didnot appear until about 1960. The graph illustrates the significantincrease in colors beginning in the 1960's, to nearly 100% by the 1980s,resulting from the almost exclusive use of color photography in morerecent years.

In FIG. 4B, there is a graph illustrating the presence of the coloryellow in photographs. Yellow can be a good indicator of aging, sinceold black and white photos printed on paper will tend to yellow overtime. Thus, from 1900 (and before) to the present, yellow exists to adegree in many photographs, with the yellow appearing in the oldestphotographs but decreasing (less yellowing in more recent photographs)until the 1960s, when more yellow appears again (due to the presence ofyellow along with other colors in modern color photos).

In FIG. 4C, there is a graph illustrating saturation in photos.Saturation typically identifies the degree or vividness of color and, asseen in FIG. 4C, saturation is not present in most photographs until the1960s, when color photos began to more widely appear. In the mid 1980s,saturation began to increase as color photography improved (early colorphotographs often had washed-out and low saturation colors, but as colorfilm technology improved and as digital photography became more common,colors became more vivid and saturated).

In FIG. 4D, there is a graph illustrating sharpness in photographs. Thischaracteristic is usually determined by the sharpness of edges in theobjects appearing in photographs, with sharpness gradually increasingfrom 1900 until about 1970 (as technical improvements in black and whitephotographs resulted in improving sharpness). Sharpness then decreasedbriefly after 1970 as color photography first became prevalent, and thenas the technology used in color photography improved (dramatically withdigital photography), the degree of sharpness in photos likewiseincreased. Sharpness can be determined by the extent of high frequencysignals in pixels at edges in digital images, is often measured as edgestrength or edge gradient magnitude, and will be more fully describedlater in conjunction with FIG. 8.

Of course, FIGS. 4A-4D are only a few possible pixel datacharacteristics among those associated with rule categories in FIG. 3.Further, many other pixel data characteristics could be evaluated beyondthose illustrated in FIG. 3. For example, certain colors change overtime because of fashion and other considerations, and thus theprevalence of one specific color over another could also be used todetermine the age of a photo. In some embodiments, objects appearing inphotos could be recognized (such as furniture, automobiles, clothing,etc.) and the color of those items in a photograph could be developedinto rules for estimating the date of a photo (along with other basicimage characteristics and pixel data characteristics).

The date estimation model 300 in FIG. 3 could use various methods andalgorithms for evaluating and weighting various characteristics based onthe analysis of the reference photographs over time (based on predictiveor probabilistic rules engines, intelligent or neural networks, or othertechniques).

In one embodiment, for any given reference date (or time frame), theimage characteristics that are likely to be present for photographs onthat date are established, based on average characteristics of thereference photos for that same date or time frame. As an example, forany given reference date or year (say 1950), each of a plurality ofimage characteristics (e.g., hues, saturation, intensity, contrast,sharpness and graininess) can have values assigned for that year basedon the observed values in the reference photos, with the observed valuescombined into an average (and discarding individual photo values thatare too far removed from the average to be reliable). Then an undatedphoto having values for its image characteristics closest to onereference date (as opposed to other reference dates) is given that yearas its estimated date.

In other embodiments, the model can be implemented as a more complex setof rules and algorithms, that assign weights to each of multiple imagecharacteristics based on the analysis of the reference photos and theirestablished dates. Values for image characteristics of an undated photoare provided to the model, and an estimated date is provided as anoutput based on the weighted characteristics built into the model rulesand algorithms. As examples only, Bayesian networks and Gaussian mixturemodels could be employed using multiple image characteristic data (asassociated with established dates of reference photos) at step 210 toclassify photos based on the observed patterns of image characteristicsover time. As examples only, various methods for pattern classificationof images based on Bayesian networks, Gaussian mixture models and otherprobabilistic and predictive techniques are described in U.S. Pat. No.7,305,132, issued to Rita Singh et al., U.S. Pat. No. 7,991,715, issuedto Jeremy Schiff et al., U.S. Pat. No. 7,961,955, issued to ThomasMinter, U.S. Pat. No. 8,050,498, issued to Gregg Wilensky et al., andU.S. Pat. No. 8,103,598, issued to Thomas Minka et al., which are eachhereby incorporated by reference.

Returning to the process in FIG. 2, when a photo is presented for dateestimation, the characteristics of the photo are identified and applied(step 212) to the date estimation model 300, and the result of theanalysis at the model 300 yields an estimated date (or time frame) forthe photo at step 214.

FIG. 5 illustrates a more detailed process for estimating the date of aphoto and, in accordance with some embodiments of the invention, foridentifying the photo with tagging information. The process of FIG. 5could be implemented at a system, such as a system 600 illustrated inFIG. 6. The system 600 receives digitally scanned photos from a scanner610, with the scanned photos stored in a database or data storage device630. As noted earlier, in some cases, photos or images may be receivedby the system 600 in digital form without having to be scanned (e.g.,photos that were originally taken with a digital camera). In the presentembodiment, family tree information is also stored in storage device630, for reasons that will be described in greater detail later.

Also seen in the system 600 is an image processing system 640 forprocessing and analyzing various data and characteristics associatedwith photos digitally scanned at scanner 610, and a rules engine 650which implements the date estimation model 300 seen in FIG. 3. A userinterface 660 permits a user to interact with the system 600, and mayinclude various input and output devices (e.g., keyboard, mouse,display, and so forth).

Returning to FIG. 5, the user of system 600 might have a stack ofhardcopy print photos which are not organized and for which identifyinginformation is missing or incomplete. The photos are scanned to providedigital copies at step 510, using a conventional scanner, such as theKODAK i600 Series Document Scanner or similar system. The scanned imagesfrom the scanner are stored in data storage device 630. As mentionedearlier, in some instances digital images may be available (rather thanprint copies of photos), and in such case, the digital photos could beinputted from a storage medium such as a compact disc, from a digitalcamera, or from other sources, such as a server storing photos at anonline photo album website.

At step 512, each image is rotated, if necessary, to put the photo inthe proper orientation. As will be discussed below, each scanned imagecan be viewed individually on a display to permit it to be associatedwith to a corresponding node of a family tree. Having the properorientation of the photo will make it easier and more efficient to vieweach photo. Software systems for determining the proper orientation of aphoto are known, and are described, for example, in U.S. Pat. No.7,215,828, issued to Jiebo Luo et al. and U.S. Pat. No. 8,036,417,issued to Andrew Gallagher et al.

At step 514, basic photo or image characteristics are determined oridentified by the system 600. Such characteristics were describedearlier and would include image border and border style, and the aspectratio of the image. In an alternative embodiment, metadata associatedwith the photo might also be identified at step 514. Metadata wouldtypically only apply to photos taken with a digital camera, and couldinclude data automatically captured at the time the photo was taken,such as the date of the photo. In other embodiments, the system 600might be configured to look for a date that might be printed on a photo(e.g., photos originally taken with film and provided as hard prints,where dates might have been put on hard prints by a photo finisher whenthe photographic film was processed). Rotation of the image at step 512facilitates the location of printed or handwritten dates, and locatingand capturing such dates (as well as other techniques for estimating ordetermining the date of photographs from basic image characteristics)are described in more detail in U.S. Pat. No. 8,036,417, issued toAndrew Gallagher et al., which is hereby incorporated by reference.

At step 520, the system then determines if data captured at step 514 hasalready established a date, thus eliminating the need for analyzingphoto characteristics (e.g., if the date is in metadata stored with adigital photo or has been captured from a date printed on the photo). Ifso, various steps in the process can be bypassed (as indicated by flowpoint A) and the photo could be sorted or categorized using the alreadyestablished date.

At step 522, the system next processes pixel data in order to identifypixel data characteristics of the photo (such as hues, saturation,intensity, contrast, sharpness and graininess), as described earlier inconjunction with FIG. 3, and using a process to be more fully describedlater in conjunction with FIG. 8. At step 524, the system applies thebasic image characteristics (from step 514) and the pixel datacharacteristics (from step 522) to the model 300 in order to determinean estimated date for the photo.

In some embodiments, as illustrated at step 530, other methods forestimating a date could also be used in addition to the analysis ofbasic image and processed pixel data characteristics. Such methods couldinclude object and event recognition (such as the model year of anautomobile, or known dates associated with events or people representedin the photos, as described in aforementioned U.S. Pat. No. 8,036,417,issued to Andrew Gallagher et al.).

As yet another example, clothing could be used for estimating a date.For example, if a photo with a known date has a person wearing aspecific item of clothing (e.g. coat, dress, hat) and that same personis wearing the same item of clothing in a second photo, it is likely thephotos have been taken in the same general time frame and the secondphoto can be given an estimated data (or timeframe) based on the knowndate of the first photo. This is particularly true for children where achild is unlikely to be wearing the same clothing beyond a one or twoyear time window. Techniques for recognizing people (based on facialrecognition), which can be used to identify a person who may be wearingthe same clothing in different pictures, are referenced later in thisdescription.

A step 532, the estimated date of the photograph is provided by thesystem. Where a photo set has been scanned and organized, the photoscould be organized by estimated date or timeframe at step 534. The usercan then view the organized photos and use the estimated dates toprovide tagging information for each photo, step 536. In otherembodiments, the photos might have estimated dates determined for eachphoto individually and tagging information provided by a user as eachdate is estimated.

The entry of tagging information for a photo according to one embodimentis illustrated in FIG. 7. A display screen 700 (as might be presented toa user at the user interface 660) includes a photo 710 for which a datehad been estimated, and a tagging window or box 720 which is used toenter tagging information. Also displayed on the screen is a family tree730, which might be stored, for example, at data storage device 630.

The family tree 730 could be created and stored in a known manner.Family tree information may include various data records for each personrepresented in the family tree, including basic information (such asname, birth date, birthplace, date of death and similar informationtaken from historical records and vital statistics), various documentsand records (including photos) that have been digitally scanned andstored in association with each person, and the relationships betweenpeople (nodes). The structure of a family tree and its representedinformation, and the manner in which various records (including imagesand photos) can be stored in association with each person or node in afamily tree, are known and are described, for example, in commonly ownedU.S. patent application Ser. No. 13/049,671, filed Mar. 16, 2011, byDonald B. Curtis, for METHODS AND SYSTEMS FOR IDENTIFICATION ANDTRANSCRIPTION OF INDIVIDUAL ANCESTRAL RECORDS AND FAMILY, U.S. patentapplication Ser. No. 13/035,816, filed Feb. 25, 2011, by Matt Rasmussen,et al., for METHODS AND SYSTEMS FOR IMPLEMENTING ANCESTRAL RELATIONSHIPGRAPHICAL INTERFACE, and U.S. patent application Ser. No. 12/511,009,filed Jul. 28, 2009, by David Graham et al., for SYSTEMS AND METHODS FORTHE ORGANIZED DISTRIBUTION OF RELATED DATA, each of which is herebyincorporated by reference.

The identification or tagging of a photo (step 536) with the use of afamily tree could be done by a user, for example, at a keyboard at theuser interface 660, and based on the estimated date of the photo. Asillustrated in FIG. 7, the system has automatically populated theestimated date of the photo at the tagging window 720. The user viewsthe family tree 730 and could infer certain things about the photo, suchas a person (name), place or event, which could then be entered as a tagby the user at the tagging window 720.

In one embodiment, where a definite association with a person at afamily tree node is made, the user might also store the photo with otherrecords at the family tree node for the recognized person. As oneexample only, a photo could be placed at a node in the family tree 730(and thus stored in data storage device 630 in association with otherdata for the person represented at the node) using a “drop and drag”technique, such as described in commonly owned U.S. patent applicationSer. No. 13/440,586, for SYSTEMS AND METHODS FOR ORGANIZING DOCUMENTS,filed on even date herewith, by Christopher Brookhart, which is herebyincorporated by reference.

Also, as an aid to the user, the system 600 could highlight or identifypeople in the family tree that were alive on the date of the photo, suchas by the use of the graphical box 750 seen in FIG. 7. In someembodiments, the system 600 could employ techniques for estimating theages of people in the photo based on the faces detected, in order toprovide an even narrower selection of people in the family tree thatmight be in the photo (based on their ages at the time of the photo).Methods for detecting faces of people in photos and estimating the agesof those people are described in commonly owned U.S. patent applicationSer. No. 13/440,811, for SYSTEM AND METHOD FOR ASSOCIATING A PHOTO WITHA DATA STRUCTURE NODE, filed on even date herewith, by ChristopherBrookhart (and various publications cited therein), which is herebyincorporated by reference.

FIG. 8 is a flow diagram illustrating the processing of pixel data atimage processing system 640 in order to identify image characteristicsand apply those characteristics to the model implemented at the rulesengine 650. At step 810, images are received at image processing system640 in the form of pixel data. In the described embodiment, colorphotographs are represented by pixel data in the RGB color space,although images could be provided in other color space models, suchCMYK, YIQ and so forth. Also, if black and white photos have beenprocessed into pixel data, the image might be provided as a gray scaleimage.

The RGB image data is then converted into the HSI (hue, saturation andintensity) color model data (or the similar HSL or HSV color models).Systems for converting RGB image data into HSI, HSV and HSL image dataare well known and described, for example, in U.S. Pat. No. 7,853,074 toGregory Mischler, U.S. Pat. No. 7,583,838 to Jae-Moon Lee, and U.S. Pat.No. 5,870,139 to Anthony Smith et al., which are each herebyincorporated by reference. It should be noted that black and whiteimages could be converted to HSI (e.g., for determining the degree ofyellowness), although for purposes of merely discerning a black andwhite photo from a color photo, the gray scale data could be retainedand applied to the rules engine 650.

The HSI image data is then separated into its various components (hue,saturation and intensity) at step 822 and the components provided to thedate estimation model 300 (as implemented by rules engine 650) in orderto estimate the age of the photo. It should be appreciated that the datacould be complex and large for each image, given that each image mighthave millions of pixels, with each pixel represented by several bytes ofdata in order to carry information on each of the components of thecolor model. In setting up the date estimation model and applying imagesto the model, average or representative values of various HSI componentsfor an entire image or portions of images could be derived and besufficient for purposes of estimating the date of a photo. In additionpixel binning and histogram analysis could be used in developing valuesfor various photographic characteristics (e.g. these illustrated inFIGS. 4A-4D).

HSI data could also be used for determining contrast (for both black andwhite images and for color images), with the model 300 built to haverules pertaining to the degree of blackness and whiteness (and graytones between black and white), weighted to account for the age of thephoto.

As the HSI color model data is identified and evaluated at steps 820 and822, non-color characteristics of the pixel data may also evaluated. Atstep 830, the RGB model data (or gray scale data for black and whitephotos) is transformed into the frequency domain using a Fourier series,generally representing the frequency of the signals derived from eachrow (or column) scan of the image. The transformation into frequencydomain data is useful to evaluate certain characteristics of the image,such as sharpness and graininess, with these characteristics generallyrepresented by signals at certain frequencies. Transforming image pixeldata into frequency domain data is known and described in the art, suchas in F. Weinhaus, ImageMagickExamples—Fourier Transforms, ImageMagickStudio LLC, http://www.imagemagick.org/Usage/fourier/, U.S. Pat. No.7,782,401, to Chih-Hsien Chou, U.S. Pat. No. 7,990,429, to Ikuya Saito,and U.S. Pat. No. 8,094,961, to Lester Ludwig.

In the described embodiment, both sharpness and graininess aredetermined for the photo to be dated. For sharpness, the edges ofobjects in an image are detected. The degree of sharpness at edges isindicated by the frequency of the component Fourier series signals atsuch edges (high frequency signals will indicate a sharp, crisp edge).Thus, by determining the presence (or lack) of such high frequencysignals, the degree of sharpness or blurriness at the edges can beidentified, as described aforementioned U.S. Pat. No. 7,782,401, toChih-Hsien Chou, U.S. Pat. No. 7,899,256, to Fedorovskaya et al., U.S.Pat. No. 7,990,429, to Ikuya Saito, and U.S. Pat. No. 8,094,961, toLester Ludwig, which are each hereby incorporated by reference. Theaverage edge sharpness or strength (edge gradient magnitude) of thephoto can be measured.

For graininess, the model determines the degree of noise (high frequencysignals) in frequency domain data across an image (rather than at itsedges), as described (as examples only) in U.S. Pat. No. 6,373,992, toKimitoshi Nagao and U.S. Pat. No. 6,801,339 to Wataru Ito, which areeach hereby incorporated by reference.

The average sharpness and the average graininess of a photo to be datedare calculated at step 832, and then applied to the date estimationmodel at step 824. Based on the application of the photo to theestimation model (and the comparison of the color and other attributesto the weighted attributes of the reference photos and their knowndates, as represented in the model), the matched attributes provide anestimated date for the photo at step 834.

While use of HSI (or HSL or HSV) color spaces/models is illustrated inthe process of FIG. 8, other forms of representing and analyzing colorcould be used. For example, color histograms could be used in developingmodel 300 and in matching photo characteristics in order to estimate thedate of a photo (at step 834).

As should be appreciated, the techniques applied to pixel data for aphotograph to be dated as described above (such as separating HSI datainto its hue, saturation and intensity components and transforming RGBdata into frequency domain data) could be used as well in analyzingreference photos for purposes of building the date estimation model 300.Further, in some embodiments, as individual photos have their datesestimated using the model 300, and particularly if an actual date isdetermined at step 520, the dates of photos (and their underlyingcharacteristics) could be provided as feedback in refining the model300.

FIG. 9 is a block diagram illustrating an exemplary computer system uponwhich embodiments of the present invention may be implemented. Thisexample illustrates a computer system 900 such as may be used, in whole,in part, or with various modifications, to provide the functions ofdatabase or data storage device 630, image processing system 640, rulesengine 650, and user interface 660, as well as other components andfunctions of the invention described herein.

The computer system 900 is shown comprising hardware elements that maybe electrically coupled via a bus 990. The hardware elements may includeone or more central processing units 910, one or more input devices 920(e.g., a mouse, a keyboard, etc.), and one or more output devices 930(e.g., a display device, a printer, etc.). The computer system 900 mayalso include one or more storage devices 940, representing remote,local, fixed, and/or removable storage devices and storage media fortemporarily and/or more permanently containing computer-readableinformation, and one or more storage media reader(s) 950 for accessingthe storage device(s) 940. By way of example, storage device(s) 940 maybe disk drives, optical storage devices, solid-state storage device suchas a random access memory (“RAM”) and/or a read-only memory (“ROM”),which can be programmable, flash-updateable or the like.

The computer system 900 may additionally include a communications system960 (e.g., a modem, a network card—wireless or wired, an infra-redcommunication device, a Bluetooth™ device, a near field communications(NFC) device, a cellular communication device, etc.). The communicationssystem 960 may permit data to be exchanged with a network, system,computer, mobile device and/or other component as described earlier. Thesystem 900 also includes working memory 980, which may include RAM andROM devices as described above. In some embodiments, the computer system900 may also include a processing acceleration unit 970, which caninclude a digital signal processor, a special-purpose processor and/orthe like.

The computer system 900 may also comprise software elements, shown asbeing located within a working memory 980, including an operating system984 and/or other code 988. Software code 988 may be used forimplementing functions of various elements of the architecture asdescribed herein. For example, software stored on and/or executed by acomputer system, such as system 900, can be used in implementing theprocess seen in FIGS. 2, 5 and 8 and the logic in the rules engine 650.

It should be appreciated that alternative embodiments of a computersystem 900 may have numerous variations from that described above. Forexample, customized hardware might also be used and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets), or both. Furthermore, there may beconnection to other computing devices such as network input/output anddata acquisition devices (not shown).

While various methods and processes described herein may be describedwith respect to particular structural and/or functional components forease of description, methods of the invention are not limited to anyparticular structural and/or functional architecture but instead can beimplemented on any suitable hardware, firmware, and/or softwareconfiguration. Similarly, while various functionalities are ascribed tocertain individual system components, unless the context dictatesotherwise, this functionality can be distributed or combined amongvarious other system components in accordance with different embodimentsof the invention. As one example, the date estimating system 600 may beimplemented by a single system having one or more storage device andprocessing elements. As another example, the system 600 may beimplemented by plural systems, with their respective functionsdistributed across different systems either in one location or across aplurality of linked locations.

Moreover, while the various flows and processes described herein (e.g.,those illustrated in FIGS. 2, 5 and 8) are described in a particularorder for ease of description, unless the context dictates otherwise,various procedures may be reordered, added, and/or omitted in accordancewith various embodiments of the invention. Moreover, the proceduresdescribed with respect to one method or process may be incorporatedwithin other described methods or processes; likewise, system componentsdescribed according to a particular structural architecture and/or withrespect to one system may be organized in alternative structuralarchitectures and/or incorporated within other described systems. Hence,while various embodiments may be described with (or without) certainfeatures for ease of description and to illustrate exemplary features,the various components and/or features described herein with respect toa particular embodiment can be substituted, added, and/or subtracted toprovide other embodiments, unless the context dictates otherwise.Consequently, although the invention has been described with respect toexemplary embodiments, it will be appreciated that the invention isintended to cover all modifications and equivalents within the scope ofthe following claims.

What is claimed is:
 1. A method for estimating an image capture date ofa photo, comprising: identifying, by a processor, image characteristicsof an undated photo; comparing, by the processor, the imagecharacteristics of the undated photo to image characteristics relatingto a plurality of reference photos, each of the plurality of referencephotos having an established date, wherein the compared imagecharacteristics comprise graininess and wherein graininess isrepresented by frequency domain data; estimating, by the processor, animage capture date of the undated photo based on a similarity of theimage characteristics of the undated photo to one or more of the imagecharacteristics of the reference photos; displaying at a user interface,by the processor, the undated photo; displaying at the user interface,by the processor, a data structure having a plurality of data nodes,each data node representing one or more data records having a subjectassociated with one or more dates; identifying at the displayed datastructure, by the processor, one or more data nodes that have arelationship to the undated photo based on the estimated image capturedate of the undated photo and the dates associated with the subject ofthe one or more data nodes; displaying, by the processor, a tagging dataentry window at the user interface for receiving tagging informationfrom a user viewing the undated photo and the displayed data structure;and receiving, by the processor, tagging information at the tagging dataentry window at the user interface concerning the undated photo based onthe identified one or more data nodes.
 2. The method of claim 1, whereinthe image characteristics are characteristics of pixel data resultingfrom digital processing of an image in the undated photo.
 3. The methodof claim 2, wherein the image characteristics further comprisesharpness, and wherein sharpness is represented by frequency domaindata.
 4. The method of claim 2, wherein the image characteristicsfurther comprise color, and wherein color is represented by an HSI (Hue,Saturation, Intensity) color model data.
 5. The method of claim 2,wherein the pixel data comprises: pixel data transformed into a Fourierseries representing frequency domain data.
 6. The method of claim 1,further comprising creating an estimation model for estimating imagecapture dates by: determining, by the processor, the imagecharacteristics of the reference photos; assigning, by the processor,the image characteristics of the reference photos to reference dates;and developing, by the processor, rules for the estimation model forassociating image characteristics of undated photos to imagecharacteristics of the reference photos, based on the imagecharacteristics of the reference photos assigned to reference dates;wherein the step of comparing the image characteristics of the undatedphoto comprises applying, by the processor, the image characteristics ofthe undated photo to the rules developed for the estimation model. 7.The method of claim 1, wherein graininess is determined by the frequencyof Fourier series signals represented across the image of the undatedphoto.
 8. The method of claim 1, wherein the data structure is a familytree, and wherein the subject of each node is a person.
 9. The method ofclaim 8, wherein the undated photo and the family tree are displayedtogether at the user interface.
 10. The method of claim 1, wherein thedata structure having a plurality of data nodes comprises a family tree,with each data node in the family tree associated with a person, andwherein the method further comprises: storing the undated photo as adata record at the one or more identified data nodes that have arelationship to the undated photo.
 11. The method of claim 1, whereinidentifying image characteristics of the undated photo comprisescalculating an average graininess of the undated photo based on thefrequency of Fourier series signals represented across the image in theundated photo.
 12. The method of claim 1, wherein the plurality ofreference photos comprise photos stored in a database system maintainingfamily trees and having information associated with dates that thestored photos were taken.